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Abstract-Today’s  systems  are  increasingly  reliant  on  software  which  must  be  sus¬ 
tained  into  the  future.  To  sustain  these  systems  organizations  must  define  sustain¬ 
ment,  meet  criteria  to  enter  sustainment,  and  overcome  some  classic  sustainment 
challenges.  This  article  discusses  these  tasks  along  with  historical  parallel  develop¬ 
ment  and  sustainment  and  potential  future  trends  in  software  sustainment. 

Introduction 

This  article  provides  an  overview  of  current  software  sus¬ 
tainment  practices  and  challenges  within  the  Department  of 
Defense  (DoD)  and  a  look  at  the  potential  future  of  software 
sustainment  within  the  federal  government.  It  takes  a  broad  view 
based  on  a  specific  study  done  in  2006  which  was  not  meant  to 
be  all  inclusive  for  every  software  sustainment  topic.  Thus  there 
are  areas  not  covered  that  may  be  relevant  to  an  individual  situ¬ 
ation.  Areas  such  as  open  source  software,  anti-tamper,  sustain¬ 
ment  cost  estimation,  and  specific  authority  and  responsibility 
for  transition  to  sustainment  should  be  explored  if  relevant  to 
your  situation. 

As  today’s  systems  become  increasingly  reliant  on  software, 
the  issues  surrounding  sustainment  become  increasingly  com¬ 
plex.  The  risks  of  ignoring  these  issues  can  potentially  undermine 
the  stability,  enhancement,  and  longevity  of  systems  in  the  field. 

At  the  center  of  this  puzzle  are  disparate  definitions.  Develop¬ 
ers  and  acquirers  have  a  general  understanding  that  sustain¬ 
ment  involves  modifying  systems  and  deploying  changes  to 
meet  customer  needs,  but  does  this  understanding  align  with 
common  practice  and  the  DoD’s  definition  of  sustainment?  DoD 
Instruction  5000.02  describes  sustainment  as  follows: 

Life-cycle  sustainment  considerations  include  supply;  main¬ 
tenance;  transportation;  sustaining  engineering;  data  manage¬ 
ment;  configuration  management;  Human  Systems  Integration 
(HSI);  environment,  safety  (including  explosives  safety),  and 
occupational  health;  protection  of  critical  program  information 
and  anti-tamper  provisions;  supportability;  and  interoperability 
[1 ,  section  8.C.1  .b]. 

The  terms  software  maintenance  and  software  sustainment 
are  often  used  interchangeably.  It  is  important  to  make  sure  that 
all  stakeholders  use  the  same  terminology  when  discussing 
software  sustainment. 

The  IEEE  Standard  Glossary  of  Software  Engineering  Termi¬ 
nology  defines  “software  maintenance”  as  follows: 

The  process  of  modifying  a  software  system  or  component 
after  delivery  to  correct  faults,  improve  performance  or  other 
attributes,  or  adapt  to  a  changed  environment  [2]. 

Software  maintenance  consists  of  correcting  faults,  improv¬ 
ing  performance  or  other  attributes,  and  adapting  to  a  changing 
organization  and  technical  environment.  To  be  complete,  there  is 
usually  a  fourth  category  of  maintenance  activities  focused  on 
anticipated  problems,  or  preventive  maintenance1  [3]. 


While  DoD  Instruction  5000.02  describes  sustainment  in  de¬ 
tail,  no  authoritative  definition  of  “software  sustainment”  exists. 
The  SEI’s  working  definition  is  as  follows: 

The  processes,  procedures,  people,  material,  and  information 
required  to  support,  maintain,  and  operate  the  software  aspects 
of  a  system. 

Given  this  definition,  software  sustainment  addresses  other 
issues  not  always  an  integral  part  of  maintenance  such  as 
documentation,  operations,  deployment,  security,  configuration 
management,  training  (users  and  sustainment  personnel),  help 
desk,  commercial  off-the  shelf  (COTS)  product  and  license  man¬ 
agement,  and  technology  refresh.  Successful  software  sustain¬ 
ment  consists  of  more  than  modifying  and  updating  source 
code.  It  also  depends  on  the  experience  of  the  sustainment 
organization,  the  skills  of  the  sustainment  team,  the  adaptability 
of  the  customer,  and  the  operational  domain  of  the  team.  Thus, 
software  maintenance  as  well  as  operations  should  be  consid¬ 
ered  part  of  software  sustainment. 

Criteria  to  Enter  Sustainment 

The  Operations  and  Support  phase  of  the  Defense  Acquisi¬ 
tion  Management  System  has  two  major  efforts,  Life-Cycle 
Sustainment  and  Disposal.  The  entrance  criteria  include  an 
approved  Capabilities  Production  Document  (CPD),  an  approved 
Life-Cycle  Sustainment  Plan  (LCSP),  and  a  successful  Full-Rate 
Production  (FRP)  decision  [1 ,  section  8.a  and  b].  In  addition,  the 
following  criteria  among  others  should  be  considered: 

•  Stable  software  production  baseline-Most  sustainment 
organizations  will  not  accept  software  into  sustainment  until 
the  software  is  stable.  Merriam-Webster  Online  defines  stable 
as  “a.  firmly  established:  fixed,  steadfast;  b.  not  changing  or 
fluctuating:  unvarying;  c.  permanent,  enduring”  [4].  However,  in 
the  realm  of  software  stable  can  mean  different  things. 

If  one  were  to  apply  Merriam-Webster’s  definition  to  soft¬ 
ware,  he  or  she  could  infer  that  a  single  instance  of  loss  of 
availability  or  a  system  failure  would  indicate  that  the  software 
is  not  stable.  In  other  words,  software  is  stable  only  if  it  does 
not  have  problems  that  cause  it  to  stop  working.  For  software, 
unfortunately,  the  definition  of  stable  can  be  a  subjective  one 
from  several  different  perspectives.  One  organization  may  be 
willing  to  accept  software  as  stable  if  it  only  fails  once  a  week, 
while  others  would  deem  this  rate  of  failure  too  high  and  would 
not  accept  the  software.  In  other  situations,  software  may 
be  considered  stable  if  no  Category  1  or  2  Software  Trouble 
Reports  (STRs)  exist. 

Defining  the  stability  of  a  system  depends  somewhat  on  its 
intended  use,  its  mission  criticality,  and  the  potential  conse¬ 
quences  if  the  system  fails.  For  instance,  a  system  such  as 
navigation  software  or  command  and  control  software  whose 
failure  could  result  in  loss  of  life  should  have  more  stringent 
requirements  for  maintaining  stability  than  one  that  is  business 
software  supporting  actions  that  could  be  postponed  for  hours 
or  even  days. 

The  program  office  should  define  the  criteria  for  accepting  a 
system  as  stable  in  the  Sustainment  Transition  Plan.  These  crite¬ 
ria  should  at  the  very  least  identify  the  types  of  STRs  allowed  to 
be  active  in  a  system  that  is  entering  sustainment. 
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•  Complete  and  current  software  documentation-Complete, 
current  software  documentation  is  paramount  for  the  software 
sustainment  organization.  Without  it,  the  sustainment  organiza¬ 
tion  has  limited  insight  into  how  the  software  was  designed 
and  implemented.  Incompleteness  or  omissions  increase  soft¬ 
ware  maintenance  costs  because  software  engineers  have  to 
reverse-engineer  the  code  to  determine  how  it  works.  In  addi¬ 
tion,  this  process  increases  the  risk  of  inadvertently  introducing 
errors  into  the  code.  Well  documented  code  is  a  plus  and-for 
those  using  incremental  and  iterative  methods-expected. 

The  program  office  should  determine  what  constitutes  com¬ 
plete  documentation  for  its  system.  At  a  minimum  the  documen¬ 
tation  set  should  include  the  “why,  how,  what,  and  where”  of  the 
system  as  built.  That  is,  documents  should  allow  the  sustainment 
organization  to  understand  why  the  system  was  designed,  how 
the  system  was  developed,  what  the  system  consists  of,  and 
where  functions  were  allocated  to  different  subsystems.  The 
overall  architecture  or  blueprint  for  the  system  needs  to  be 
provided.  Plans  on  how  the  program  office  intended  to  handle 
COTS  and  configuration  management  issues  are  essential  for 
sustainment  and  continued  implementation.  Interface  definitions 
need  to  be  documented.  Database  designs  and  their  docu¬ 
mentation  are  essential  to  understanding  their  purpose  within 
the  system.  Lastly,  the  development  environment  needs  to  be 
defined  so  the  sustainment  organization  knows  what  tools  were 
used  to  develop  and  support  the  system. 

•  Authority  to  Operate  (ATO)  for  an  operational  software 
system- Before  a  system  can  be  considered  operational  in  the 
field  and  thus  meet  the  criteria  to  enter  sustainment,  an  Author¬ 
ity  to  Operate  must  be  issued.  The  ATO  issuance  depends  on 
approval  of  security  requirements  by  the  Designated  Approval 
Authority  (DAA).  Issuing  an  ATO  means  that  a  DAA  has  accept¬ 
ed  that  operation  of  the  system  represents  a  low  security  risk.  An 
ATO  is  issued  for  a  fixed  period  of  time  (typically  three  years)  and 
must  be  renewed.  Delay  in  obtaining  ATO  approval  or  renewal 
could  cause  the  system  to  be  deemed  non-operational. 

•  Current  and  negotiated  Sustainment  Transition  Plan-In 
many  instances,  a  program  has  been  developed,  tested,  and 
declared  operational  but  there  is  no  funding  set  aside  to  address 
creation  and  subsequent  negotiation  of  the  Sustainment  Transi¬ 
tion  Plan.  Unfortunately,  in  an  era  where  budgets  are  becoming 
increasingly  tight,  sustainment  planning  is  postponed  and  in 
some  instances  forgotten. 

Both  the  development  organization  and  the  sustainment 
organization  need  to  be  involved  in  creating  the  Sustainment 
Transition  Plan.  If  a  contractor  is  involved  in  development,  that 
organization  also  needs  to  participate  in  the  development  and 
subsequent  negotiation  of  the  Sustainment  Transition  Plan.  In 
addition,  the  contract  should  include  tasks  that  address  the  con¬ 
tractor’s  role  in  the  sustainment  planning  and  transition  process. 

The  program  office  should  ensure  that  while  the  program  is 
being  developed,  sustainment  tasks  are  not  forgotten  or  removed 
from  the  development  contractor’s  tasking.  While  the  development 
contractor  may  not  necessarily  become  the  sustainment  organiza¬ 
tion,  the  development  contractor  is  responsible  for  developing  and 
maintaining  documentation  that  the  sustainment  organization  will 
need.  It  is  the  program  office’s  responsibility  to  ensure  that  the  con¬ 


tractor  does  not  create  documentation  that  is  proprietary  or  unde¬ 
liverable.  Even  though  it  was  cancelled  in  1998,  the  MIL- STD -49 8, 
Section  5.1 3,  “Preparing  for  Software  Transition,”  contains  good 
background  and  reference  material  in  this  area  [5]. 

•  Sustainment  staffing  and  training  plan-Staffing  the  sustain¬ 
ment  organization  is  critical.  The  staff  needs  to  be  trained  software 
professionals  that  can  work  with  the  development  organization 

to  transfer  the  necessary  system  knowledge.  One  should  not  as¬ 
sume  that  any  of  the  development  organization  staff  will  transition 
to  the  sustainment  organization;  rather,  adopt  a  plan  to  transfer  the 
knowledge  from  one  organization  to  the  other  as  part  of  the  staff¬ 
ing  plan.  The  staffing  and  training  plan  are  related  to  and  should 
be  coordinated  with  the  Sustainment  Transition  Plan. 

As  with  many  other  areas  associated  with  sustainment,  train¬ 
ing  for  the  sustainment  organization  is  often  treated  as  an  after¬ 
thought  and  is  usually  an  under-funded  activity.  Even  though  the 
sustainment  staff  is  composed  of  trained  software  professionals, 
they  still  need  training  on  the  specifics  of  the  system  entering 
sustainment.  This  is  especially  true  for  the  increasingly  complex 
systems  that  contain  a  mixture  of  COTS,  government  off-the- 
shelf  (GOTS),  and  organic  (government-developed)  software 
code.  “On-the-job”  training  is  not  sufficient  for  personnel  sus¬ 
taining  these  types  of  complex  systems.  The  system’s  specific 
architecture,  design  decisions,  and  other  nuances  need  to  be 
communicated  in  some  depth. 

Sustainment  Challenges 

Our  research  in  the  2006  timeframe  identified  a  variety  of 
issues  or  challenges  prevalent  with  software  sustainment  at  that 
time.  These  were  grouped  into  six  categories.  This  is  not  to  say 
that  one  would  not  find  other  issues  that  must  be  addressed 
when  a  system  is  entering  sustainment.  In  addition,  no  priority  is 
implied  by  the  order  in  which  these  topics  are  discussed. 

The  following  categories  of  sustainment  challenges  were  identified: 

•  Sustainment  with  COTS  software-requires  consideration  of 
system  obsolescence,  technology  refresh,  source  code  escrow, 
and  vendor  license  management  among  related  topics. 

•  Programmatic  considerations-discusses  issues  with 
relegating  the  sustainment  requirement  to  the  category  of 
“minor  requirements’’ 

•  System  transition  to  sustainment-considers  topics  of  support 
database  transition,  development  and  software  support  environ¬ 
ment  infrastructure  (software  test  lab,  hardware  spares,  release 
processes  and  procedures),  staffing,  operations  training,  and 
transition  planning 

•  User  support-discusses  help  desk,  user  documentation,  and 
user  training. 

•  Information  assurance-discusses  the  unique  challenges  of  IA 
and  COTS  software  products  and  testing  for  IA. 

•  Development  versus  sustainment. 

While  these  challenges  are  most  likely  still  valid,  the  only  one 
discussed  in  depth  within  this  article  is  the  last  one,  development 
versus  sustainment.2 

Parallel  Development  and  Sustainment-History 

As  I  found  in  2006  (and  continuing  to  the  present),  many 
systems  are  fielded  in  an  incremental  manner.  Incremental 
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means  that  an  increment  or  version  of  the  system  that  pro¬ 
vides  partial  capabilities  is  developed  and  fielded.  The  remain¬ 
ing  capabilities  are  developed  later  depending  on  budget, 
requirements  definition,  and  technology  advancements.  For  the 
sustainment  organization,  this  means  that  it  will  be  sustaining 
a  system  in  parallel  with  another  version  of  the  system  that  is 
still  underdevelopment. 

Development  in  parallel  with  sustainment  is  not  a  new  con¬ 
cept;  however,  many  sustainment  organizations  may  not  have 
experience  with  this  mode  of  operation.  In  some  instances, 
upgrades  (development)  are  considered  a  sustainment  activity. 
This  makes  the  “line”  between  development  and  sustainment 
very  hazy.  To  ensure  continued  operation  of  the  system,  the 
sustainment  and  development  organizations  need  to  develop 
processes  and  procedures,  coordinate  them  with  all  parties,  and 
obtain  concurrence  on  their  use.  This  should  include  an  under¬ 
standing  of  who  is  responsible  for  any  upgrades. 

Historically,  in  organizations  that  are  successful  in  performing 
development  and  sustainment  concurrently,  groups  within  the 
organizations  report  to  the  same  person.  Given  the  organiza¬ 
tional  structure  of  the  development  and  sustainment  organiza¬ 
tions,  this  can  be  problematic.  In  many  instances,  the  person 
who  has  enough  experience  to  oversee  both  the  development 
and  sustainment  groups  does  not  have  the  desire  or  the  time  to 
be  involved  in  this  level  of  oversight. 

To  better  align  parallel  development  and  sustainment  efforts, 
the  program  office  needs  to  consider  the  current  sustainment 
structure.  With  that  in  mind,  it  should  then  determine  how  the 
system  being  developed  is  evolving  and  how  it  can  fit  into  the 
sustainment  structure.  Sustainment  organizations  should  plan  to 
adapt  their  processes  to  handle  an  evolving  system,  especially  if 
it  implements  COTS  hardware  or  software  products. 

In  addition,  a  joint  (development  and  sustainment)  Configura¬ 
tion  Control  Board  (CCB)  needs  to  be  created  and  given  the 
authority  to  act.  All  decisions  for  changes  to  the  baseline  must 
go  through  the  CCB  without  exception.  The  operational  soft¬ 
ware  must  be  driven  from  the  CCB  approved  baseline.  Last,  a 
clear,  documented  path  of  escalation  up  to  senior-level  person¬ 
nel  must  be  created  to  address  issues.  It  is  not  a  question  of  if 
there  will  be  issues,  but  when  they  will  occur.  Being  prepared  to 
handle  issues  reduces  the  impact  problems  have  on  the  overall 
development  and  sustainment  of  the  system.  Emphasis  in  bold 
is  added  to  point  out  the  criticality  of  following  a  strict  CCB 
process  when  there  are  two  organizations  (one  development 
focused  and  one  sustainment  focused).  Otherwise,  keeping  the 
two  systems  in  alignment  will  be  problematic  at  best. 

Future  of  Software  Sustainment 

In  2006,  when  I  authored  the  Sustaining  Software  Intensive 
Systems  technical  note,  many  commercial  organizations  were 
starting  to  use  incremental  and  iterative  methods  known  as 
Agile  methods.  These  methods  have  evolved  over  time;  today 
the  commercial  environment  is  using  something  referred  to  as 
DevOps.  What  is  DevOps? 

What  it  is.  A  way  of  working  that  encourages  the  Develop¬ 
ment  and  Operations  teams  to  work  together  in  a  highly  col¬ 
laborative  way  towards  the  same  goal. 


What  it  is  not.  A  way  to  get  developers  to  take  on  operational 
tasks  and  vice  versa  [6]. 

Strangely,  I  find  the  definition  very  similar  to  what  was  de¬ 
scribed  in  the  Parallel  Development  and  Sustainment  section. 
However,  there  are  some  major  differences.  DevOps  seems  to 
be  the  Agile  community’s  term  for  doing  sustainment  and  opera¬ 
tions  in  parallel.  The  methods  used  are  based  on  the  Agile  Mani¬ 
festo  four  tenets  and  1 2  principles  but  applied  in  a  sustainment 
environment.  Adopting  these  tenets  and  principles  within  DoD 
requires  a  major  change  in  the  paradigm  for  doing  business  [7]. 

The  SEI  currently  has  a  team  researching  the  use  of  Agile 
methods  in  sustainment  within  the  federal  government.  This 
research  is  how  I  came  upon  the  term  DevOps.  In  addition,  Gene 
Kim  provided  a  keynote  speech  on  DevOps  at  the  2013  Soft¬ 
ware  Technology  Conference.  The  question  is  whether  this  type 
of  methodology  will  be  useful  and  adopted  within  the  federal 
government.  We’re  still  trying  to  determine  this.3 

However,  we  have  learned  that  several  maintenance  organiza¬ 
tions  within  the  federal  government  are  trending  toward  using  more 
Agile-like  methods  for  conducting  sustainment.  While  the  “jury  is 
still  out”  on  whether  Agile  methods  are  indeed  in  use,  there  seems 
to  be  a  movement  to  try  more  incremental  and  iterative  methods 
using  empowered  teams.  This  movement  toward  incremental  and 
iterative  methods  does  seem  to  make  sense  for  a  sustainment 
environment  where  defects  and/or  enhancements  are  prioritized 
and  worked  on  in  that  order  based  on  the  amount  of  capacity  the 
sustainment  team  possesses.  This  approach  sounds  eerily  like  the 
product  backlog  maintained  by  an  Agile  team  [8]. 

In  fact,  one  of  the  early  conclusions  by  SEI  in  our  Agile  work 
included  the  following  thoughts  on  using  Agile  for  sustainment: 

Operations  and  Support  is  where  sustainment  of  the  software 
is  conducted.  It  is  assumed  that  the  software  previously  devel¬ 
oped  (during  the  Engineering  and  Manufacturing  Development 
phase)  is  mature  and  stable,  so  the  anticipated  software  effort 
expended  during  this  phase  is  low  and  should  follow  a  sustain¬ 
ment  model,  driven  by  the  need  to  correct  errors  observed  during 
qualification  testing,  or  providing  enhancements  as  requested  by 
program  stakeholders.  It  is  quite  possible  for  a  software  develop¬ 
ment  team  working  in  these  life  cycle  phases  to  follow  an  Agile 
approach.  Quite  often  the  features  requested  during  this  phase 
are  modifications  that  are  only  relevant  within  the  context  of  the 
system  that  had  been  previously  developed.  The  aspect  of  user 
involvement  that  naturally  occurs  at  this  point  of  the  life  cycle 
makes  it  easier  for  the  use  of  a  collaborative  approach. 

It  should  be  noted  that  some  of  the  Agile  methods  might  not 
be  as  practical  as  others4  during  the  Operations  and  Support 
phase.  For  example,  it  is  quite  likely  that  the  capability  provided 
during  sustainment  is  planned  to  be  provided  over  a  significant 
period  of  time,  typically  on  the  order  of  two  years.  While  the  in¬ 
volvement  of  the  user  might  be  beneficial,  the  frequent  releases 
may  not  be  useful  because  of  limitations  with  the  verification 
and  validation  environments  required  for  deployed  systems.  On 
the  other  hand,  this  constraint  should  not  preclude  the  use  of 
Agile  during  this  stage  of  development  [8]. 

In  addition,  many  issues  need  to  be  explored  including  but  not 
limited  to  documentation  required,  CCB  interaction,  release  of 
updated  software  to  the  field,  quality  of  code,  and  cost  of  code. 


CrossTolk-January/February  2014  35 


LEGACY  SYSTEM  SOFTWARE  SUSTAINMENT 


Our  ongoing  Agile  and  sustainment  research  is  looking  at  these 
and  other  issues.  The  results  of  our  Agile  and  sustainment  study 
should  be  available  in  early  2014. 

Summary 

There  are  multiple  issues  associated  with  software  sustain¬ 
ment.  They  start  with  agreeing  on  a  standard  definition  for  the 
term  software  sustainment.  This  is  followed  by  knowing  the 
criteria  for  entering  sustainment  which  include  stable  software 
production  baseline;  complete  and  current  software  documenta¬ 
tion;  Authority  to  Operate;  current  and  negotiated  Sustainment 
Transition  Plan;  and  sustainment  staffing  and  training  plan.  Finally, 
specific  known  challenges  need  to  be  considered.  These  include 
but  are  not  limited  to  sustainment  with  COTS  software;  program¬ 
matic  considerations;  system  transition  to  sustainment;  user  sup¬ 
port;  information  assurance;  and  development  versus  sustainment. 

Parallel  development  and  sustainment  have  historically  been 
done  which  may  lead  to  a  move  towards  the  more  current  DevOps 
approach.  DevOps  is  becoming  popularized  by  the  Agile  move¬ 
ment.  Many  issues  need  to  be  resolved  and  the  jury  is  still  out  on 
the  effectiveness  of  this  approach  in  the  federal  government.^ 
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1.  Information  for  sustainment  is  based  on  the  SEI  Technical  Note  Sustaining  Software-Intensive  Systems, 
CMU/SEI-2006-TN-007  and  updated  to  reflect  the  DoD  I  5000.02  released  in  2008 
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